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(54) Packet network transit delay measurement system 



(57) A system for analyzing and estimating average 
transit delay and high percentiles of delay in a packet 
network (10) uses detailed packet switch measure- 
ments (30) that are normally stored in a network data- 
base for maintenance and engineering purposes. The 
system randomly selects source (12) and destination 
(14) points and samples the stored maintenance meas- 
urements (30) for network circuit elements located be- 
tween the randomly selected source and destination 



points. These sampled measurements are transmitted 
to the system processor via electronic mail and the sys- 
tem generates a single server queuing model (40) to de- 
termine average and variance of delay for each compo- 
nent. A network-of -queues model is then constructed to 
determine the end-to-end average delay and end-to- 
end variance of delay. An Erlang probability distribution 
is then applied to these end-to-end values to provide 
statistics of packet transit delay in the packet network 
(10). 
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Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to data communictions networks, and more particularly to a system for measurina 
transit delay in a packet network. M 

Description of the Related Art 



In many data commun.cations networks, data is generated in high intensity bursts that are often separated by idle 
periods therebetween. The average data rate may, therefore, be much lower than the peak rate. An economic way of 
transmitting this type of intermittent (bursty) data is to assemble the data into what is known in the art as "packets" and 
interspersing packets from several different channels on a single, physical communication path. This communication 
path is therefore occupied by a particular packet only for the time corresponding to the packet length, and the com- 
munication path is then available for use by other packets that may be between different data terminals 

A packet refers to a group of bits that are switched in the network as an integral unit. A packet may also include 
a header that .s used to identify that packet. A packet is formatted in a particular way and may contain data and 
*° destination, origination and control information. 

In a typical packet-switching network, a number of users can be interconnected. Packet switches are utilized in 
the network to sort packets received from one circuit and direct these packets to another circuit according to the header 
information for each packet. A typical packet switch is AT&T's 1 PSS™ packet switch which implements the CCITT X 
25 packet-switching protocol. 

25 m any data communications network, a primary concern in the performance of the network is the time in which 

information is processed and transmitted back to the user. This time is typically referred to as response time or network 
transit delay. Network transit delay in a packet-switching network may be a function of data transmission speed prop- 
agation time, distance through the network, packet-switch processing time and queuing delay. 

To evaluate the performance of a packet network, transit delay is generally measured from the time the last bit of 

■*> a particular packet enters the provider's originating central office equipment (e.g., originating packet switch) to the time 
the first bit of the packet leaves the provider's destination central office equipment (e.g.. destination packet switch) 

in designing and mainta.n.ng data communications systems, it is essential to minimize network transit delays In 
fact, most customers today have strict requirements regarding the permissible transit delay in the network Customers 
will often require the provider of the data communications network to pay a penalty if the transit delay exceeds a 

j5 predetermined level. 

Therefore, it is important for the design and maintenance of a data communications system to effectively measure 
transit delay in the packet network. Moreover, customers of the system frequently require periodic measurements of 
transit delay to ascertain whether the system is performing within the limits of the customers' service contracts 

There are currently a number of measurement systems used today to estimate or measure network transit delay. 
These measurement systems generally fall into two categories and are illustrated in FIG. 1. 

The first type of measurement system is a protocol analyzer 20 which includes hardware connected to the packet 
network 10 for analyzing and measuring transit delay. Transit delay is measured by connecting a protocol analyzer 20 
to a desired measurement point in the network 10 and essentially time stamping a particular packet to determine the 
actual transit delay of that particular packet. 

This type of measurement system, however, can be extremely expensive because it requires a protocol analyzer 
20 at each desired measurement point. Thus, if a customer has numerous access points to the network 10 as they 
often do, protocol analyzers 20 would be required at each point of access into the network 10. 

In addition to the expense associated with requiring a protocol analyzer 20 at each desired measurement point 
each protocol analyzer 20 must be located in the service node (i.e., the packet switch) in the network 10 and also • 
requires space for locating the protocol analyzer 20, power for operating the equipment, and the ability to access the 
equipment for installation and maintenance purposes. 

The second type of system for measuring transit delay is through the use of measurement software 22 that may 
exist in the customer's end equipment and/or at various locations in the packet network 10. The measurement software 
22, however, must be developed, tested and integrated into the network 10 and frequently requires updating (reinte- 
gration) each time the software in the packet network 10 is updated, changed or otherwise modified (software main- 
tenance). 

It is, therefore, desirable to provide a cost effective, non-invasive measurement system to estimate transit delay 
in a packet network 10 that does not require hardware or measuring software to be added to the network 10. 
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SUMMARY OF INVENTION 

The present invention provides a cost effective system for estimating the end-to-end packet delay distributions in 
a packet network. The system analyzes and estimates average transit delay and high percentiles of delay in a packet 

s network by utilizing detailed packet switch measurements that are normally collected and stored in a network database 
for maintenance and engineering purposes. The system randomly selects various source and destination points and 
samples the stored maintenance measurements for specific network circuit elements between the randomly selected 
source and destination points. These sampled measurements are transmitted to the system processor and a single 
server queuing model is generated to determine average and variance of delay for each identified element. A network- 

10 of-queues model is then constructed to determine the end-to-end average delay and end-to-end variance of delay. An 
Erlang probability distribution is then applied to these end-to-end values to provide statistics of packet transit delay in 
the packet network. 

BRIEF DESCRIPTION OF THE DRAWINGS 

is 

FIG. 1 is a network overview illustrating the prior art methods of measuring packet network transit delay. 
FIG. 2 is an illustration of the measurement modelling approach of the present invention. 
FIG. 3 is an illustration of a preferred embodiment of the present invention. 

FIG. 4 illustrates the network-of-queues model utilized in the preferred embodiment of the present invention. 
20 FIGS. 5-7 are block diagrams of steps performed in the preferred embodiment of the present invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The present invention provides a cost effective method for estimating the end-to-end delay distributions in a packet 
2S network. Referring to FIG. 2, a packet network 10 is generally shown connecting a data source 1 2 to a data destination 
14. Although not illustrated, it is commonly known that a packet network 10 may consist of numerous elements or 
components, including, but not limited to, access lines, gateway links, inter-switch trunks : packet switches, packet 
assembler and disassemblers (PAD) ; concentrators, multiplexers and interfaces. 

As part of the normal operation of a packet network 10, it is common for a provider of the network 10 to collect 
30 certain packet switch maintenance measurements or data 30 for engineering and maintenance purposes to keep the 
network 10 operational. The maintenance measurements 30 are typically stored in a database 32 of network 10 as 
illustrated in FIGS. 2 and 3. The maintenance measurements 30 also represent the amount of packets that are trans- 
mitted through the network 10. 

For each circuit in the network 10, these maintenance measurements 30 commonly include packet counts, byte 
35 counts, speed of circuit, errored frame rate, average packet size, packet retransmission rate, packet switch utilization 
and circuit utilization. Although well known in the art. the above-mentioned maintenance measurements 30 will be 
generally described below. 

A packet count generally refers to the number of packets switched by a particular element (i.e., a packet switch) 
in the network 10 in a given period of time. Byte count refers to the number of bytes contained in the information field 
-to of the packets that have been switched by the particular element in the network 10 in a specified interval of time. 

Speed of circuit is a fixed parameter that generally refers to the capacity of the network circuit in bits per second. 
Errored frame rate refers to the number of frames (data link layer) that were received in error in a given interval of time. 

Average packet size generally refers to the average size of packets measured in bytes. Packet retransmission 
rate refers to the number of packets retransmitted (e.g., transmitted more than once) in the network 10 in a given period 
of time. 1 

Packet switch utilization refers to the fraction of the capacity of the packet switch processor (not shown) that is 
utilized in a given interval of time. Packet switch utilization is based upon the previously described packet count. 

Circuit utilization refers to the fraction of the capacity of the network circuit (not shown) that is utilized in a specified 
period of time. Circuit utilization is derived from the previously described byte count., 
so As is illustrated in FIGS. 2 and 3 : the present invention utilizes the above-mentioned maintenance measurements 

30 for specific network elements to generate a model 40 of the network 10. The network model 40 is utilized to estimate 
the end-to-end delay in the network 10. 

The sequence of operation of the preferred embodiment of the present invention is illustrated in FIGS. 5-7. The 
maintenance measurements 30 in network 10 are collected (step 100) and. stored (step 110) in database 32. As pre- 
ss viously stated, steps 100 and 110 are commonly performed in packet networks 10. 

Since a typical packet network 10 may be connected to multiple source points 12 and destination points 14, it is 
necessary to identify a particular source point 12, destination point 14. date and time of measurement in order to 
determine the packet network transit delay (step 120). This identification may be accomplished by user input 
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In the preferred embodiment of the present invention, however, it is preferable to randomly sample different points 
in the network 10. The preferred embodiment, therefore, randomly selects a desired source point 12 and destination 
point 14 for measuring the network transit delay therebetween, and randomly chooses the date and time of such 
measurement. 

By randomly selecting the source and destination points 12,14, and date and time of sampling, the present inven- 
tion is capable of generating a model 40 of the network 10 and need not take actual measurements between every 
source and destination point 12, 14 connected to the network 10. This is desirable because, for every N end points in 
network 10, there exist N2 end-to-end points. Thus, if there were 5000 end points in network 10. there would be 25 
m.lhon paths to measure. Assuming that a customer desired hourly network transit delay measurements (also referred 
to as quality of service (QOS) measurements), this would amount to approximately 20 billion QOS measurements per 
month. Selecting random end-to-end paths based upon randomly selected source and destination points 12 14 and 
taking measurements at random times dramatically decreases the number of QOS measurements necessary to Gen- 
erate the model 40 of the network 10. 3 

Although the preferred embodiment of the present invention samples the packet network transit delay between 
randomly selected source and destination points 1 2, 14, it is understood.that this method is utilized to make the system 
more practical. However, it is not a prerequisite of the present invention to randomly sample network transit delay. 
Wh.le it may be unpractical or inefficient, the present invention may also be utilized to actually measure transit delay 
based upon each source and destination point 12, 14 connected to network 10. 

Once the desired source and destination points 12, 14 are identified in step 120, the system identifies the packet 
switches in network 10 that each end point is connected to (step 130). The network packet switches are determined 
based upon the particular source and destination points 12. 14 previously identified in step 120 

In step 140, the system also identifies the trunks and tandem switches (if any) between the previously identified 
source and destination points 12,14. Having determined the desired source and destination points 12 1 4 in step 1 30 
the present invention utilizes known routing tables in step 140 to identify the trunks and tandem switches in the network 
10 between the source and destination points 12. 14. Thus, for each desired source and destination point 12 14 the 
system identifies the trunks and associated packet switches in network 10 connecting the desired source and desti- 
nation poinjs 12, 14. 

Having identified the associated packet switches in network 10 that connect the desired source point 12 to the 
des.red destination point 14, the system then reads (step 150) the relevant maintenance measurements 30 collected 
and stored in the network database 32 (steps 100, 110) for each identified packet switch corresponding to the identified 
source and destination points 12,14, transports (step 1 60) the relevant maintenance measurements 30 to the modelling 
processor 50, and feeds (step 170) the relevant maintenance measurements 30 into the network modelling program 
operating in the modelling processor 50 as illustrated in FIG. 3. Methods well known in the art to transfer such data 
include, but are not limited to, electronic mail systems or file transfer protocol. 

By utilizing the relevant maintenance measurements 30, the modelling processor 50 estimates end-to-end delay 
distributions for packet network 10. The system generates a model 40 of the network 10 by generating single server 
M/G/1 queuing models for individual network elements (e.g., identified packet switches) and constructing a network- 
of-queues model in a series as illustrated in FIG. 4. For analytical tractability all queues are assumed to be independent 
As will be discussed below, an Erlang probability distribution is then fit to two end-to-end delay parameters (average 
and variance of delay) to predict service delay performance of the network 10. 

Thus, for each network element (N), the system generates a network element single server queuing model Each 
element model has two primary output parameters: average transit delay (m f ) and variance of packet delay (a,) The 
average transit delay (m f ) is broken down into two components, namely, a constant (non-variable) component (m ■<=) 
and a variable component (nry). The variable component (mf) is derived from the randomly variable portion of the 
45 packet delay. 

Referring to step 180 in FIG. 6, the average transit delay (m,) for each network element (N) is calculated using the 
following well known formula for an IWG/1 queuing model: 

0 + C J) 
m '=^ (1+ ^2(TT^) 

where x- represents average service time. Pj server utilization, a xj 2 variance of service time, and C K - 2 =o 2/fx.)2 (the 
coefficient of variation). Xl 1 

Referring now to step 190 in FIG. 6, the variance of packet delay (aft is determined using the following well known 
formula for an M/G/1 queuing model: 
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where x/ 3 represents the third moment of service time, Xj average service time, p; server utilization, a xi 2 variance of 
service time ; and C bj 2 =a xj 2 /(Xj) 2 (the coefficient of variation). 

Once the average transit delay (m^ and variance of packet delay (Oj 2 ) are determined for each network element 
(N) in steps 180 and 190, end-to-end performance measures for network 10 are obtained in step 200 by addition of 
each mj-and o^ 2 resulting in end-to-end average delays (both fixed (m c ) and variable (m v )) and end-to-end variance 
of delay (o 2 ). 

The details of step 200 are given below: Let nrij and Oj 2 denote the average and variance of delay of the ith network 
element (N). Let mf and m^ denote the constant and variable portions of the average delay respectively Accordingly, 
the end-to-end average (m) and variance (a 2 ) packet delay for the network 10 is determined as follows: 
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Once step 200 is completed and the end-to-end average packet delay (m) and variance of delay (a 2 ) are determined 
for network 10, a simple Erlang probability distribution- is applied to the end-to-end average and variance of delay in 
step 210, thereby providing the high percentiles of delay (e.g., 95th percentile) frequently required by customers. Only 
the variable portion of the average delay (m v ) and the variance of delay (a 2 ) are used in the Erlang distribution. The 
fixed portion of the average delay m c is excluded to avoid biasing the fit through effects of propagation delay and 
constant componentsof processing or insertion delays. The bias tends to make the Erlang distribution have high order 
which may cause under-estimation of the high percentile of delay. 

The Erlang model requires two parameters, namely average and order (r). The average is simply set to m v : while 
theorder(r)=min[max[1 : L(m v /a) 2 J],10]. Theorder(r) is limited to values between 1 and 10. The value of 1 corresponds 
to an M/M/1 queue, while the maximum value of 10 ensures that a certain minimum variability is maintained in the" 
model 40. 

The resulting Erlang distribution as a function of delay (d) is denoted E(m v ,r ; d). The pth percentile of delay corre- 
sponding to a given probability (p) is denoted E~ 1 (rn v r.p). The network delay performance measures are then simply 
estimated as follows: 
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Average Delay: m 

pth Percentile of Delay: E'VnrV.rpHm 0 . 

The table below shows the relationship among E-^m^r.p), rand p for various example values of rand p. .Note that 
for given values of r and p, E' 1 (m v ,r,p) is proportional to m v . 
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Once the high percentiles of delay are determined in step 21 0 ; the system outputs the results as illustrated in step 
220 of FIG. 7. In the event that all data has not been read (step 230) : the system repeats steps 150 through 220 until 
all of the desired data has been processed. When all of the data has been read in step 230. the system generates a 
complete QOS summary report (step 240) detailing the performance of the network 10 with respect to its packet transit 
delay objective. 

It is to be understood that the above description is only of a preferred embodiment of the present invention Nu- 
merous other arrangements may be devised by one skilled in the art without departing from the spirit and scope of the 
invention. The invention is thus limited only as defined in the accompanying claims. 
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Claims 



1. A packet network transit delay measuring system for estimating packet transit delay in an existing packet network 
having a plurality of packet switches, comprising the steps of: 

30 

(a) collecting packet switch measurements. 

(b) determining element packet delay statistics based upon said collected packet switch measurements: 

(c) determining end-to-end packet delay statistics from said element packet delay statistics: 

(d) randomly selecting a source point and a destination point to sample packet transit delay therebetween: 
35 ( e) identifying the packet switches located on a circuit between the randomly selected source and destination 

points; and 

wherein the element packet delay statistics are determined for each identified packet switch based upon col- 
lected measurements for that identified packet switch. 

A system as claimed in claim 1 , further comprising the step of storing the collected packet switch measurements 
in a network database 
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3. A system as claimed in claim 2, wherein said collected packet switch measurements are transmitted from said 
network database to a processor means for determining element and end-to-end packet delay statistics. 

4. A system as claimed in claim 1 . wherein the packet switches located on the circuit between the randomly selected 
source and destination points are identified using routing tables. 

5. A system as claimed in claim 3 S wherein said collected packet switch measurements are transmitted from said 
network database to said processor means by electronic mail. 

6. A system as claimed in claim 3, wherein said collected packet switch measurements are transmitted from said 
network database to said processor means by file transfer protocol. 

s5 7. A system as claimed in claim 1 . wherein the determination of said element delay statistics for each identified packet 
switch comprise the steps of: 

(a) determining average packet transit delay based upon said packet switch measurements: and 
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(b) determining variance of packet delay based upon said packet switch measurements. 

8. A system as claimed in claim 7. wherein the determination of said end-to-end delay statistics comprise the step 
of summing the average packet transit delay and variance of packet delay for each identified packet switch. 

5 

9. A system as claimed in claim 8, wherein high percentiles of delay statistics are determined by applying an Erlang 
probability distribution to the end-to-end delay statistics. 

10. A system for estimating packet transit delay in an existing packet network having a plurality of packet switches 
10 said system comprising: 

(a) means for collecting and storing maintenance measurements for said packet switches: 

(b) processing means for determining element and end-to-end packet delay statistics based upon said [packet 
switch] collected and stored maintenance measurements; and 

is (c) means for transmitting said stored maintenance measurements from said collecting and storing means to 

said processor; and wherein said processing means: 

(d) randomly selects a source point and a destination point to sample packet transit delay therebetween: and 

(e) identifies the packet switches located on a circuit between the randomly selected source and destination 
points. 

20 

11. A system as claimed in claim 10. wherein the element packet delay statistics are determined for each identified 
packet switch based upon collected measurements for that particular identified packet switch. 

12. A system as claimed in claim 10, wherein the packet switches are identified using routing tables. 

25 

13. A system as claimed in claim 12. wherein said element delay statistics comprise: 

(a) average packet transit delay for each identified packet switch determined from the transmitted measure- 
ments for each particular identified packet switch: and 
30 (b) variance of packet delay for each identified packet switch determined from the transmitted measurements 

for each particular identified packet switch. 

14. A system as claimed in claim 1 3, wherein said processor determines said end-to-end delay statistics by summing 
the average packet transit delay and variance of packet delay for each identified packet switch. 

35 

15. A system as claimed in claim 1 4, wherein said processor determines high percentiles of delay statistics by applying 
an Erlang probability distribution to the end-to-end delay statistics. 
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FIG. 5 
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